Multi label vs. Multi class
An expriment in loss functions on the same dataset.
- Tank classifier models:
- Bulding a model using multi class loss
- Bulding a model using multi label loss
- Addtional validation (out of train scope images)
- Comparison summary
tank_types = 'merkava mk4','M1 Abrams','water'
path = Path('tanks')
#downloading 150 images to labeled directories
if not path.exists():
path.mkdir()
for o in tank_types:
dest = (path/o)
dest.mkdir(exist_ok=True)
urls = search_images_ddg(f'{o} tank', max_images=150)
download_images(dest, urls=urls)
#deleting not working files
fns = get_image_files(path)
failed = verify_images(fns)
failed.map(Path.unlink);
tanks = DataBlock(
blocks=(ImageBlock, CategoryBlock),
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parent_label,
item_tfms=RandomResizedCrop(224, min_scale=0.5),
batch_tfms=aug_transforms())
dls = tanks.dataloaders(path)
dls.valid.show_batch(max_n=5, nrows=1)
learnMC = cnn_learner(dls, resnet18, metrics=error_rate)
learnMC.fine_tune(4) #fastai already chooses multi class loss becuase od the CategoryBlock
Review top 10 loss images:
def parentlabel(x):
return [x.parent.name] # as get_y recieve a list
tanks2 = DataBlock(
blocks=(ImageBlock, MultiCategoryBlock),
#MultiCategoryBlock(add_na=True)
get_items=get_image_files,
splitter=RandomSplitter(valid_pct=0.2, seed=42),
get_y=parentlabel,
item_tfms=RandomResizedCrop(224, min_scale=0.5),
batch_tfms=aug_transforms())
dls2 = tanks2.dataloaders(path)
dls2.valid.show_batch(nrows=1, ncols=5)
learnML = cnn_learner(dls2, resnet18, metrics=accuracy_multi)
#defaults for the accuracy multi are: threshold=0.5, Sigmod=True
learnML.fine_tune(4)
Addjusting the threshhold (as the curve is smooth we shuold nit be worried about overfitting):
preds, targs = learnML.get_preds()
xs = torch.linspace(0.05,0.95,29)
accs = [accuracy_multi(preds, targs, thresh=i, sigmoid=False) for i in xs]
plt.plot(xs,accs);
learnML.metrics = partial(accuracy_multi, thresh=0.85)
learnML.validate()
Review top 10 loss images:
The first model is binary and forces to select a single class based on any image input. Its softmax encourages a single class-selection even more. While the second model is multi-label and predicts each class separately, it may be better at alerting images outside the scope of the class or two classes within one image.
Now let's see if one model can perform better on out-of-scope images even though they were both trained on the same dataset:
twotanks = PILImage.create('/content/mek_abs_3.jpg')
twotanks.show()
learnMC.predict(twotanks)
learnML.predict(twotanks)
bicycle = PILImage.create('/content/bycycle_nature.jpg')
bicycle.show()
learnMC.predict(bicycle)
learnML.predict(bicycle)
piano = PILImage.create('/content/Grand-Piano.jpg')
piano.show()
learnMC.predict('/content/Grand-Piano.jpg')
learnML.predict('/content/Grand-Piano.jpg')
Although we used two different loss functions which are designed for a different purpose, based on the tests, it turns out both models are performing quite similar:
Metrics - both models are around a 7% error rate (actually adjusting the multi label threshold increases its accuracy but I believe the difference is more connected to process randomness and different metric functions than the models). Top loses - looks like the models are having problems on the same images. Out of scope train images - small random check, the multi label model didn't show any advantage in identity the images were out of scope or label two different images as we might expect. The bottom line proves that the main factor is the data set, the model will be able to learn as good as the examples he learn during training and as we didn't train the multi label model with the relevant examples, in this case, it couldn't out perform the multi class model.